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28 June 1973 


25X1 


MEMORANDUM FOR: 

SUBJECT: 

REFERENCE: 


Displaying Uncertainty to Decisionmakers 

Memo from A. W. Marshall, 15 June 1973, 
subject as above, DCI/IC 73-0579 


1. Regarding your note attached to Andy Marshall's memo 
referenced above, this is a matter which has been discussed at 
one level or another in the intelligence community for more than 
ten years to my own knowledge, and some experimentation has taken 
place, but the results are not thusfar impressive. The President's 
memorandum of November 1971 gave a new impetus with its charge that 
"a determined effort be made to upgrade analysis personnel and 
analysis methods." 

2. An introduction to quantitative methodologies is provided 
intelligence personnel in training courses conducted by CIA and 
DIA, and DIA has an in-house effort going with contractor support, 
focusing primarily on application of Bayes' theorem. 

3. We attempted last year to get a PRG effort going on this 
problem, to no avail, but perhaps the time is ripe to form a working 
group and charge it with coming up with some recommendations for 
application standards and/or a community R&D effort which will result 
in a set of accepted probability applications and/or methods. 


4. What we ought to be looking for is tools which would be 
useful in the analysis process--recognizing at the same time that 
obtaining the cooperation and support of analysts and production 
office managers who are not quantitative-minded is going to be a 
continuing problem. 


5. PRG will work on a^specific proposal 


” (\ 


Attachment 

reference 


* 


Ljfcputy Chief, PRG 

* — lO)^ 


Approved For Release 2004/08/30 ffiMftgOO 

i A *• "A a A 


25X1 



sgfc 25X1 

. , 25X1 



■ pj" i 

ETW3 

I 



t/ 25X1 





aii t &jBB ~ .L; -. t'i id^^mkrSiM 

o/^Z 


Approved For R«tease 2004/08/30 : CIA-RDP80M01133AQP08001 10003^. — , 

i- . ' ^ ?9 l 


NATIONAL SECURITY COUNCIL 

WASHINGTON, D.C. 20506 

June 15, 1973 


MEMORANDUM FOR 


V / ADM. V. P. dePOIX 
M/GEN. LEW ALLEN 
M/GEN. D. O. GRAHAM 
B/GEN. SAMUEL V. WILSON 
EDWARD W. PROCT OR 

I ^ 25X1 

FR OM: A. W. MARSHALL 


SUBJECT: 


Displaying Uncertainty to Decisionmakers 


As you know, I have been working with several individuals 
in efforts to experiment with the use of probabilities and 
related mathematical tools in community products. Attached 
is a paper prepared for me by Tom Brown of RAND on one 
aspect of the subject. You might find it interesting and 
informative. I remain interested in this method of 
analysis -- especially in improving the communication of 
appropriate levels of uncertainty in intelligence judgments. 
Current methods are unsatisfactory in my view. 
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DESIRABLE WAYS OF DISPLAYING UNCERTAINTY 

TO 

DECISIONMAKERS 


T. A. Brown 
May 4, 1973 


1 . Why Should Explicit Probabilities be Used ? 

Probability is a language which has been worked out over the past three 
or four hundred years to express varying degrees of uncertainty; it is 
routinely used in science, insurance, and sports; it is used by systems 
analysts, hard scientists, actuaries, accountants, demographers, 
gamblers, soldiers, and a growing body of social scientists; although 
some of the theory of probability involves difficult abstract mathematics 
it is not necessary to master the theory in order to use the language 
meaningfully. One would certainly expect, therefore, that there would be 
substantial use of probability language in a field as filled with uncertainty 
as intelligence. By attaching probabilities to alternative ultimately 
confirmable statements about the world, an intelligence analyst is able to 
express his inner state of certainty (or uncertainty) with brevity and 
precision, in a form which can be scored for accuracy (in the long run) 
and thus contribute to his track record. The purpose of this note is to 
suggest some patterns, formats, and techniques for expressing intel- 
ligence assessments in probabilistic terms which will be easily grouped 
by the consumer while preserving the inherent potential advantages of 
brevity, precision, and scorability. 

2. What Should Not be Displayed to the Decisionmaker . 

Probability theory is a fairly mature brarch of mathematical science. It 
is rich in concepts (such as Bayes' Theorem) and devices (such as eve nt 
trees and state diagrams) designed to assist in the calculation of prob- 
abilities. An assessor is naturally tempted to expose the ingeniotis and 
interesting techniques he has employed in deriving his assessments to 
the appreciative eyes of the decisionmaker. This temptation must be 
firmly resisted or the brevity of communication will be lost. An event- 
tree (probability diagram) which fills a page takes longer to absorb than 
a page of text, and must usually be accompanied by many pages of 
definition of the states involved and justification for the probabilities 
ascribed to the various branches. This material is useful to other 
analysts, perhaps, in discovering how and where differe nces hr — - 
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probability assessments arise, but it is no more use to the decisionmake 
than would be the worksheets used by the analyst in working out a purely 
verbal forecast. 


The analyst should be careful that the alternatives to which he ascribes 
probabilities are themselves precisely defined. A probability distri - 
bution over vague statements adds up to just another vague statement. 

If care is not taken in this regard, the statements made to the decision- 
maker will be neither precise nor scorable. 

A third temptation which must be avoided is the use of excessively '’trick's 
methods of display. It is often possible, by selecting proper scales or 
format, to pack a tremendous amount of information into a single picture. 
What you gain in information density, however, you tend to lose in ease 
of communication. Therefore, formats should be standardized as much 
as possible so that the decisionmaker can instantly gr<St£p what is be ng 
said. This is especially true for probabilistic statements, where it mav 
be expected that the analyst will be more at home with the language "han 
the decisionmaker. 

3. How Probabilities Should be Expressed . 

There are an infinite number of mathematically equivalent ways of ex- 
pressing a probability. We shall discuss six of these, each of which hat 
its own peculiar advantages, and evaluate which of them are most 
appropriate for probabilistic statements to decisionmakers. 

The simplest way to present a probability is to simply state it as a 
probability. The most obvious advantage of this is simplicity, directness 
and the wide understanding of what the word means; a more subtle advan- 
tage is that if you are stating the probabilities of more than two disjointed 
but exhaustive alternatives it is easy to normalize (i. e. , the probabilities 
must add up to 1); the normalization condition on other methods of 
presentation is much more complex. 

Another way of reporting a probability is to give the negative of the 
logarithm of the probability. This is the amount of "information" in the 
event. If the logarithm is to the base two, it is expressed in bits. This 
quantity is called the amount of information in the event because if you 
have a communication device which has a fixed cost for transmitting a 
zero or a one (that is, a fixed cost per bit), then you will minimize your 
expected cost of communication by selecting a code which assigns to eacn 
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Name 

Symbol 

Range 

F ormula 

Advantages 

Probability 

P 

OSp^l 

P 

Common use; 
easy norma Lizas; on 

Log Probability 

I 

0<l£c-a 

1= -log 2 p 

. Information theory; 
scoring system 

Odds 

d 

OSdS 00 

d=...E... 

1 -p 

Common uEe; 
wagers 

Log Odds 

L 

-oPfL too 

L - log d 

Bayes' theorem 

Pari -mutual Payoff 

M 



Use at race tracks 

Insurance Premium 

P 

Of pi 1 

*d 

ii 

A probability is 
a price 


Six Equivalent Ways of Expressing 
a Probability 


TABLE 1 


Approved For Release 2004/08/30 : CIA-RDP80M01 133A0008001 10003-6 


Approved For Release 2004/08/30 : CIA-RDP80M01133A0008001 10003-6 


Probability 

Information Odds 

Log Odds 

Parimutual Payoff 

Premiur 

0 

4 c=0 

0 

_ oo 

4 e*o 

( 

. 05 

4. 32 

. 053 

-1. 28 

20. 

05 

. 1 

3. 32 

.111 

- .95 

10. 

1 

. 2 

2. 32 

. 25 

- . 60 

5. 

2 

. 3 

1. 74 

. 43 

- .37 

3. 33 

, 3 

. 4 

1. 32 

. 67 

- . 18 

2. 5 

. 4 

. 5 

o 

o 

1. 00 

0 

2. 

. 5 

. 6 

. 74 

1. 5 

. 18 

1. 67 

. 6 

.7 

. 51 

2. 33 

. 37 

1.43 

. 7 

. 8 

. 32 

4. 

. 60 

1. 25 

. 8 

. 9 

. 15 

9. 

.95 

1. 11 

. 9 

. 95 

. 07 

19. 

1. 28 

1. 05 

. 9f 

1. 

0 

4 crO 

4 e=«0 

1. 

1 . 


A Short Table of Values of Alternative Ways 
of Expressing a Probability 



TABLE 2 
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of the possible events you may have to report a signal containing a number 
of bits equal to the negative log2 of the probability of the event. In other 
words, if you have selected an optimal code, -log 2 p is the number oi bits 
you will have to send to communicate the occurrence of an event whose 
probability is p. The reader familiar with reproducing scoring systems 
will recognize the logarithm of the probability as a scoring system which 
encourages a respondent to be "honest" (i. e. , to report his true subjects < 
probability that a given event will take place). These characteristics 
make the log of .the probability a useful training device for analysts and 
of great interest to information theorists, but I suspect it would tend to 
confuse a decisionmaker who is not very familar with these disciplines. 

A more down-to-earth way of expressing a probability is to translate it 
into the equivalent betting odds. The higher the odds, the more likely the 
event is to occur. If the odds on an event are 2:1, then any chance to bet 
on the event at odds of 1.9*1 should be accepted. A bet at odds at 2.1:1 
should be rejected. Some individuals are very comfortable thinking in 
terms of odds. I understand from one person with a great deal of ex 
perience teaching probability theory to analysts that many individuals find 
odds repugnant (perhaps because of their association with frivolous games 
of chance) while they find probabilities (which have a much more "scientit 
aura) perfectly acceptable. 

Some Bayesians (notably Ward Edwards) have advocated encoding prob- 
abilities by means of the logarithm of the odds. The advantage of this is 
that under Bayes' Theorem the change in the log odds ratio if a new 
piece of evidence comes in depends only on the new evidence, not on 
the prior value of the log- odds ratio. To be perfectly explicit, let us 
suppose that we have a prior probability p that hypothesis H is true (and 
thus probability 1-p that H is not true). Suppose an event E takes place, 
with: 

p(E/H) = probability that E occurs if H is true. 

p(E/^H)= probability that E occurs if H is not true. 

Theorem the posterior odds will be: 

X P< E /H) 
p(E/~H) 


then by Bayes 1 

_e_ 

1-p 
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Thus the log odds ratio will change by an amount 

lo|! 

regardless of what its initial value may have been If vou haw •+ • . 

a ™ ' 1 lS n °Vt Pt be ° alled UP ° n t0 per£ °™ computational .nan paU- 

is not aoorom at * hil * that this P^rticuiar method of tncJu, 

ot appropriate for communicating with the decisionmaker. 

beta.vmf 7 ° f T C °, d i ng Pr ° babUitieS is to ca lculate the amount you w . Jd 

Thia s , f°“ 3 1° Wln a $ 1 • 00 for a bet placed with a bookie on th, .went 
, simply the reciprocal of the probability of the event Fn- ■ ) i 

plac^aTrace tracks^” Th^ decisi ° nnaakin g under explicit uncertainty takes'^ S 
place at race tracks, this quantity (perhaps multiplied by two) will provide 

quite a meaningful expression for the likelihood of a given event U tt,- 

decisionmaker in question falls into this category, then the use it thi . 

encoding may have something to recommend it. 

A final way of encoding a probability is to consider it a price- the nr nun, 

t y a°ki„Tlce aVe T h ° Pay £ ° r 3 ° ne - d0Uar Policy against^e^eu 

event so T h^ price is numerically equal to the probability of the 
event, so you might argue that this encoding is simply identical to the 

a thB Pr0bab i“ ty Ba ‘ WchoLgicillvlt s 

a little different. Viewing a probability as a price (an idea due I bel < 
to Savage) makes it obvious that it makes sense to speak of the probatiiifv 
of inherently unique events. Many people, who have been exposed to r bit 
positivist epistemology, hold that to speak of a probability^ mua , 

probabimv) o° me i SUb ' Se » ° f a SGt ° f eqUaUy likely ° aSeS (combinational 
.p bability) or else an inherently repeatable case (such as flipping a c~m 

for example). Calling a probability a price helps this sort of person 

overcome their inhibitions against applying the concepts in the context of 

elections resolutions, wars, economic change, and all the other miq r 

events which characterize the world of the national security decisionmaker. 

le enC ? ding SGhemeS aWe » ^t us consider identically 

the same piece of information encoded in six different ways: 7 


'The probability that Broyhill will be re-elected is . 7. 


rr 


Approved For Release 2004/08/30 : CIA-RDP80M01 133A0008001 10003-6 


Approved For Release 2004/08/30 : CIA-RDP8OMO1133AO0O8OO1 10003-6 

7 


"The information content of Broyhill's re-election would be 
about half a bit. " 

"I will lay $2. 33 to $1. 00 that Broyhill is re-elected. " 

rr The log odds (base 10) on Broyhill's re-election are + . 37. " 

"A two dollar bet on Broyhill's re-election pays $2. 86. " 

"A $1, 000 insurance policy against Broyhill's re-election 
costs $700. " 

Which of these is to be preferred? I like the first and last best: they a 
the most straight-forward. Next I like odds, and after that the belting 
house pay-off. The encodings involving logarithms are too esoteric for 
the decisionmaker unless he intends to do some computation, which see: 
highly unlikely to me. 

4. "Futures" Charts . 

The futures market in any commodity is a predictive device. It is a 
forecast of what the price of a given commodity is going to be at some 
specific date in the future. Anyone interested in the commodity will 
study the futures market and the movement of future prices with great 
care, for it indicates a consensus of views on the probable supply and 
demand over the immediate future. 

Couldn't we set up a similar mechanism for forecasting events other 
than the movement of commodity prices? For example, a market in 
insurance policies that General Flup will (or will not) be premier of 
Ruritania on December 31, 1973? If we would try to set up an actual 
market, the administrative burden of keeping track on who sold what 
and who bought what would outweigh the benefits of the operation, but 
we could set up an excellent simulacrum of such a market by exploiting 
reproducing scoring systems. For example, if we had three experts on 
Ruritania we could ask each of them to produce by each Monday noon an. 
estimate of the probability the General Flup will be premier of Ruritamn 
on December 31, 197 3, together with a brief explanation of what factors 
in the situation have caused them to change their estimate from the 
preceding Monday. At the end of the year each of the three experts 
would be individually rewarded proportionally to the sum of the scores 
of their weekly forecasts (note: it is important not to give any "sp'ecial" 


ns 
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reward to the forecaster achieving the highest score, as this would 
disrupt the incentive system and tend to induce excessively "risky” 
estimates). If a crisis flares up, the experts could, of course, begin 
revising their estimates daily or even hourly. 

Making such estimates on a systematic schedule, and graphing them up 
appropriately (see Figure 3) has many advantages. First, when suc i a 
chart has been established and the decisionmaker has taken the trouble 
to understand it once, he will be able to absorb its message very rapidly 
whenever he looks at it in the future. Second, the chart automatically 
generates a chronology of relevant events and indications and gives a 
subjective grading of their relative importance (an important event 
corresponds to a big jump in the graph: an unimportant one to a small 
jump). Third, graphs such as these are very "scorable" (generating, 
as they do, fifty or more responses from each analyst on each question). 
Fourth, they provide a ready vehicle for expressing "minority opinion" 
by analysts. For example, in the chart shown (Figure 3) one analyst did 
not drop the probability of General Flup remaining in office in response 
to reports of back-country violence, while the others did. The minority 
analysts' assessment is plotted separately from the mean of the major it/ 
for a time. This is quite an advantage of quantitative assessments of 
uncertainty: where a disagreement ai'ises, it is very easy to report in 
just how much it amounts to and report it precisely. There is no need 
to waste time trying to create some vague verbal formula which covers 
all points o' view or to draft reclamas. Furthermore, since each analys 
is held re sponsible for his own assessments at scoring time there is 
motivation to dissent when and only when you feel you are right and the 
majority is wrong (if the assessments are prepared independently, as th« 
ought to be, the "dissent" is probably not the right word; the important 
point is that since the analysts are each individually responsible for the : 
individual assessments, rather than being held collectively responsible 
for a group assessment, individual deviations will occur fairly freq aentq 
and will be ultimately punished or rewarded as they are less or mm e 
accurate than the group as a whole). 

Another example of such a graph may be found on page 23 of Peterson, 
Kelly, Barclay, Hazard Handbook for Decision Analysis : Inference 
from Evidence : Bayes 1 Theorem . This example is presented to 
illustrate the application of Bayes 1 Theorem, but such a display is 
useful as a communication device whether you use Bayes or some other 
statistical technique or just informed intuition as the basis for makmg 
your forecasts. My personal prejudice is that the informed intuition of 
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Graph of "Event Futures" - Probability General Glup will 
will be Premier of Ruritania on December 31, 1973 


FIGURE 3 


*One analyst feels that the back-country violence is not serious, and 
indeed has no political significance. His estimate is indicated by 
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a person properly trained in using the language of probabilities will 
usually turn out to be the most sensitive and accurate instrument 
available in arriving at probability estimates of real-world events; but 
this is an empirical matter, and it may turn out in real cases that 
people who use formal methods consistently out perform those who do 
not. 

Some questions on which you may want to keep futures charts are 
difficult to express as probabilities of single events. For example, let 
us suppose you wish to forecast the GNP of the United States for 1973. 

You could cast this as "What is the probability that the GNP will exceed 
1. 3 trillion dollars in 1973? " but this format would clearly lack 
resolution. A better format would be to ask each analyst to estimate 
(say) the median, upper quartile, lower quartile estimate of what the 
GNP will turn out to be. If he makes such an estimate each week, then 
you can grade him at the end of the year by using some recipe to con- 
struct a distribution from the three numbers he gave you and applying 
one at the reproducing scoring systems appropriate to continuous dis- 
tributions. The distributions generated by different analysts may be 
combined in any of a number of ways, and as in the case of single event 
probabilities any distribution which varies too much from the "combined ' 
distribution may be displayed separately. The graph would look some- 
thing like Figure 4. 

Some will complain that keeping charts of this kind is apt. to turn out, ir 
many cases, to be rather a waste of time because something which looks 
like a hot issue or big problem when the chart is set up may recede from 
interest before the chart reaches completion. Well, if this happens the 
completion of the chart will be a very mild burden on the analyst (if 
General Flup dies, for instance, the analyst hardly has to pore over 
the dispatches from Ruritania to decide what his chances at the premier- 
ship might be). In any case, a single analyst could be responsible for 
providing inputs to 10 or 12 charts each week. If an average of three 
analysts are feeding data to each chart, then this means that we will be 
able to keep 3. 5 to 4 charts going for each analyst assigned to this duty. 
Thus 50 analysts ought to provide enough brain-power to keep 175 - 200 
charts going. 

5. Point Estimates . 

Almost any intelligence report which contains an element of prognostication 
can be recast in explicit probabilistic terms. If you take all the implicit 
probabilistic forecasts in a paper, make them explicit, and simply list 
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1, 400 Billion 

Upper Quartile 

1, 350 

Est. Median 

1, 300 Billion 

Lower Quartile 

1, 250 Billion 

1, 200 Billion 

JFMAM JJASOND 

Consensus of Probabilistic Estimates of U.S. GNP 
for 1973 

FIGURE 4 
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them, this will often turn out to constitute an excellent executive summuy 

an easfwaTL V Paper ' Having P^P«ed such a summui v 

an easy way to ask some group other than the authors of the paper lor 

comments is to simply ask them what probabilities they would aserbe 
to the various events in question. 

If you do this in a systematic manner, you will soon (in a year or so) 
begin to get some objective measure of the relative accuracy of CIA, 

, and any other intelligence service you have available. I think it 
is very dangerous for a decisionmaker to evaluate his intelligence services 
on the grounds of whether they structure the problem the way he does or ' 
not. If he is right and the intelligence service is wrong a couple of times 
he may easily get carried away by hubris and commit some really big 
errors. Intelligence should be rated for accuracy and relevance, not 
for how well it fits the decisionmaker's view of the world. 
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